| Imputation | Bias (CR) | Coverage (CR) | Bias (SR) | Coverage (SR) |
|---|---|---|---|---|
| normal (-3, -3) | -0.53 | 0.998 | -1.12 | 0.492 |
| normal (0, 0) | -0.12 | 1.000 | -0.71 | 0.886 |
| normal (3, 3) | 0.27 | 1.000 | -0.32 | 0.998 |
| tmvn (1, 5) | -0.04 | 1.000 | -0.01 | 1.000 |
| uniform (0, 0) | -0.16 | 1.000 | -0.75 | 1.000 |
EMPG Meeting, Padova, 3 September 2025
A statistical technique to quantitatively summarize the results of multiple studies to derive a more precise effect estimate.
Various sources of heterogeneity: clinical, methodological, statistical (Higgins et al. 2024).
“[…] many clinical studies have more than one outcome variable; this is the norm rather than the exception. These variables are seldom independent and so each must carry some information about the others. If we can use this information, we should.” (Bland 2011)
Many psychological studies report multiple outcomes (Tyler, Normand, and Horton 2011). Multivariate meta-analysis (MVMA):
allows for a joint estimation (Riley et al. 2017)
handles outcomes missing “by design” (Jackson et al. 2017)
Rubin (1976) introduced a foundational framework distinguishing missing data mechanisms: Missing Completely At Random (MCAR), Missing At Random (MAR), Missing Not At Random (MNAR)
These mechanisms differ from missing data patterns, which refer to the observed structure of which values are present or absent.
Patterns = “What” is missing
Mechanisms = “Why” it is missing
MCAR: Missingness is unrelated to observed or unobserved data. The missing sample is a random subsample.
\[ P(M \mid Y, \phi) = P(M \mid \phi) \]
MAR: Missingness depends on observed variables.
\[ P(M \mid Y, \phi) = P(M \mid Y_{obs}, \phi) \]
MNAR: Missingness depends on unobserved variables (or both).
\[ P(M \mid Y, \phi) = P(M \mid Y_{mis}, \phi) \]
\[ P(M \mid Y, \phi) = P(M \mid Y_{obs}, Y_{mis}, \phi) \]
Multivariate meta-analysis with sensitivity analysis
Pre-existing work on imputation of summary-level missing data (Carpenter, Kenward, and White 2007; Fiero, Hsu, and Bell 2017; Lu 2023; Pastore et al. 2017; Viechtbauer 2022)
Replace the unobserved outcomes with many plausible draws, then meta-analyse each imputed dataset and pool the results with Rubin’s rules.
mixmeta)Missing outcomes were drawn from one of three distributions:
Each condition was repeated 10,000 times, with:
mixmeta (Sera 2019)| Imputation | Bias (CR) | Coverage (CR) | Bias (SR) | Coverage (SR) |
|---|---|---|---|---|
| normal (-3, -3) | -0.53 | 0.998 | -1.12 | 0.492 |
| normal (0, 0) | -0.12 | 1.000 | -0.71 | 0.886 |
| normal (3, 3) | 0.27 | 1.000 | -0.32 | 0.998 |
| tmvn (1, 5) | -0.04 | 1.000 | -0.01 | 1.000 |
| uniform (0, 0) | -0.16 | 1.000 | -0.75 | 1.000 |
Bias depends on missingness rule
Imputation from uniform distributions are conservative but imprecise
Imputation from truncate multivariate normal best performance, but relies on true model knowledge
In real data multiple assumptions should be tested
Cuijpers et al. (2010) collected data from 48 studies that measure depression on both a clinician rating (HRSD; Hamilton (1960)) and self-report scale (BDI; Beck et al. (1961)). The meta-analysis highlights a substantial difference between the patients’ and clinicians’ evaluations of depression, in favor of the clinician rating.
37 studies from Appendix A (HRSD-17 and BDI outcomes, single control group, data from METAPSY database)
3 studies lacked post-treatment data
For multiple comparisons per study:
I make explicit my assumptions regarding missing data, by exploring different scenarios.
library(missmeta)
library(tmvtnorm)
library(mvtnorm)
unif11 <- function(n) runif(n, min = -11, max = 11)
norm06 <- function(n) rnorm(n, mean = 0, sd = 6)
norm66 <- function(n) rnorm(n, mean = 6, sd = 6)
normn66 <- function(n) rnorm(n, mean = -6, sd = 6)
sigma <- matrix(c(6^2,
CorCov(6,6,0.7),
CorCov(6,6,0.7),
6^2), nrow = 2)
lower <- c(-100, -100)
upper <- c(100, 100)
mtv1 <- function(n) rtmvnorm(n, mean = c(-6, -6), sigma = sigma, lower = lower, upper = upper)[,1]
mtv2 <- function(n) rtmvnorm(n, mean = c(-6, -6), sigma = sigma, lower = lower, upper = upper)[,2]Using the function genimp we generate 500 imputed values from each distribution.
imp1 <- list(unif11, norm06, norm66, normn66, mtv1)
imp2 <- list(unif11, norm06, norm66, normn66, mtv2)
out <- mapply(function(i1, i2) {
genimp(
df = dmnar,
iter = 500,
imp1 = i1,
imp2 = i2,
eff1 = "EstCR",
eff2 = "EstSR",
se1 = "SECR",
se2 = "SESR",
cor = "Cor.ws",
N = "N",
imprho = 0.6
)
}, i1 = imp1, i2 = imp2, SIMPLIFY = FALSE)Afterwards, we perform the chosen analyses on each imputed dataset. The user can autonomously choose the preferred meta-analysis package to conduct multivariate meta-analysis (e.g., metaSEM, mixmeta, metafor).
library(mixmeta)
outls = unlist(out, recursive = FALSE)
res = lapply(outls, function(data) {
theta = cbind(data$EstCR, data$EstSR)
Sigma = cbind(data$SECR^2, CorCov(data$SECR, data$SESR, data$Cor.ws), data$SESR^2)
m = mixmeta(theta, S = Sigma, method = "ml")
s = summary(m)
ci = confint(m)
results = data.frame(
eff1 = s$coefficients[1, 1],
eff2 = s$coefficients[2, 1],
se1 = s$coefficients[1, 2],
se2 = s$coefficients[2, 2],
cov12 = s$vcov[1, 2],
ci.lb1 = ci[1, 1], ci.ub1 = ci[1, 2],
ci.lb2 = ci[2, 1], ci.ub2 = ci[2, 2]
)
})
res <- do.call(rbind, res)
res <- split(res, rep(1:5, each=500))With makepooldata we prepare the dataset to make it suitable for pooling estimates and standard errors obtained from the iterations using Rubin’s rules.
pooldata <- lapply(res, function(x) {
makepooldata(data = x, effs = "eff", ses = "se", covs = "cov")
})
methods <- c("Uniform (-11, 11)", "Normal CR = 0, SR = 0", "Normal CR = 6, SR = 6",
"Normal CR = -6, SR = -6", "Multivariate Normal (-6, -6)")
sumres <- mapply(function(p, label) {
sumeth_multi(Q_mat = p$Q_mat, U_list = p$U_list, method = label)
}, p = pooldata, label = methods, SIMPLIFY = FALSE)Via a simulation study and a real dataset we have shown the use of a flexible tool for imputing missing outcome data that:
Forces the researcher to make explicit the assumptions on the missing outcome data
Allows for imputation and robustness check of results under different scenarios
Can handle MNAR data
Is easy use and with missmeta quite straightforward to implement
Results depend on the imputation distributions and parameters
Performance can drop if assumptions are far from reality (SR)
Slow at the moment if one has to test many assumptions
Increase computational efficiency
Extend to more than two outcomes in multivariate meta-analysis
Explore performance against existing imputation methods and sensitivity analysis techniques